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ABSTRACT 

This  paper  describes  the  development  and  testing  of  Nonlinear  Display  Mapping  (NDM),  which  is  high- 
speed digital  signal  processing  to  quickly  manipulate  displayed  images.  The  identification  performance  of 
observers  using  an  automated  algorithm  and  observers  using  NDM  are  compared.  The  observer  utilizes 
NDM  to  improve  image  understanding.  No  algorithm  or  automatic  method  can  optimize  thermal  displays 
for  every  environment,  condition  and  target.  Often,  the  manual  controls  are  used  when  “auto  mode”  does 
not  work  well.  The  commonly  available  manual  brightness  and  contrast  controls  are  difficult  to  use  and  do 
not  fully  realize  the  potential  of  digital  systems.  Also,  recent  experiments  showed  that  the  auto  mode  might 
combine  contrast  shades  such  that  targets  or  target  features  are  hidden  from  the  observer.  Current  display 
algorithms  employ  a wide  variety  of  methods,  including  histogram  equalization,  local  area  processing,  and 
region  of  interest  processing.  Non-linear  Display  Mapping  (NDM)  differs  from  these  because  it  allows  the 
user  to  manipulate  the  displayed  intensity  of  different  regions  of  the  sensor  output  by  real-time  non-linear 
mapping  to  pixel  values.  The  user  can  thus  allocate  or  “tune”  pixel  intensities  (gray  shades)  to  output 
regions  expected  to  contain  targets.  This  avoids  squandering  the  system’s  limited  dynamic  range  on  image 
features  such  as  cold  sky,  clouds,  trees  or  water.  In  other  words,  NDM  enables  the  user  to  tune  the  sensor  to 
the  scene. 


1.  INTRODUCTION 

The  Night  Vision  Electronic  Sensing  Directorate  Modeling  Branch  studied  various  sensor 
effects  in  the  late  1990’s  through  still-image  degradation  experiments.  The  effects  of  blur, 
sampling  noise,  spurious  response  and  other  variables  on  target  identification  were 
studied  in  experiments  that  used  a standard  set  of  12-target  12-aspect  high-resolution  12- 
bit  thermal  images  as  a baseline1.  Once  degraded,  these  images  had  to  be  optimized  for  8- 
bit  displays  and  it  was  found  that  no  single  technique  could  consistently  deliver  good 
contrast  across  the  entire  set  of  images  while  retaining  critical  target  information.  A novel 
method  of  non-linear  mapping  was  devised  that  allowed  the  image  processor  to  view  the 
image  histogram  and  determine  non-linear  bit  allocations  for  display  based  on  observable 
target  features.  Any  target  image  could  be  quickly  processed  to  achieve  50%  contrast 
through  this  method. 

Thermal  sight  users  would  benefit  from  such  target  contrast  improvements  if  they  could 
be  achieved  at  a reasonable  cost.  Software  was  developed  to  simulate  a simple  non-linear 
sensor  control.  Over  900  high-,  medium-  and  low-resolution  thermal  images  with  over  50 
targets  and  6 different  backgrounds  were  processed  for  expert  analysis.  That  analysis 
suggested  that  while  non-linear  mapping  could  improve  target  contrast  for  almost  all 
studied  conditions  and  targets,  it  would  be  most  beneficial  under  conditions  of  moderate 
to  high  blur  and  moderate  to  low  target  resolvability.  When  blur  is  low  and  resolution 
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high,  littie  improvement  is  possible  because  the  observer  has  adequate  target  information 
to  ID,  regardless  of  target  contrast.  In  conditions  of  very  high  blur  and  very  low 
resolution,  the  target  becomes  an  amorphous  blob,  and  it  is  not  possible  to  distinguish 
target  features  under  any  circumstances.  Non-linear  contrast  enhancements  are  thus  most 
beneficial  between  those  extremes. 

A low-resolution  highly  blurred  target  set  with  a predicted  probability  of  ID  under  30% 
tests  the  extreme  limit  of  non-linear  mapping’s  potential  to  improve  image 
comprehension.  The  B-kit  automatic  algorithm  emulator  was  chosen  for  comparison 
because  it  is  scene  based,  known  to  produce  relatively  good  target  contrast  under  most 
conditions,  and  similar  algorithms  are  incorporated  into  2nd  generation  FLIR  systems. 

Night  Vision  Electronic  Sensors  Directorate  researchers  conducted  a forced-choice 
vehicle  ID  experiment  in  which  20  trained  subjects  were  shown  low  resolution  images 
processed  with  a B-kit  automatic  algorithm  emulator  and  with  Non-linear  Display 
Mapping  (NDM)  techniques.  Subjects  viewed  50  images  processed  with  each  method 
presented  in  a randomized  order.  The  mean  probabilities  of  identification  for  B-kit 
automatic  and  NDM  were  25%  and  32%  respectively.  Nineteen  of  20  observers  had 
higher  scores  with  NDM,  and  the  improvement  range  was  12%  to  100%.  The  average 
improvement  was  28%. 

2.  BACKGROUND  AND  PURPOSE 

The  2nd  generation  Forward  Looking  Infrared  (FLIR)  imaging  systems  currently  being 
developed  and  fielded  have  digital  data  output,  whereas  1st  generation  FLIR’s  have 
analog  output.  Analog  display  controls  generally  allow  the  user  to  manipulate  the 
brightness  and  contrast  of  the  image,  but  such  controls  affect  the  entire  data  output  signal, 
(generally  8 bit),  and  can  introduce  additional  noise  that  obscures  comprehension.  While 
displays  are  often  still  limited  to  8 bits,  the  output  signal  of  the  2nd  generation  FLIR  is 
twelve  bits  or  more  and,  because  it  is  digital,  can  be  mapped  to  the  display  in  numerous 
ways.  Thus,  regions  of  the  12-bit  output  signal  are  accentuated,  or  de-emphasized  or 
eliminated  when  mapped  to  the  8 -bit  display. 

Automatic  algorithms  and  methods,  such  as  region  of  interest  and  local  area  processing 
are  useful  image  comprehension  aids  in  some  conditions,  but  the  dynamic  variability  of 
thermal  scenes  and  targets  ultimately  confounds  these  methods.  The  automatic  method  is 
not  capable  of  prioritizing  displayed  gray  shades  based  on  subjective,  situational  or 
qualitative  scene  information.  For  example,  significant  gray  shades  may  be  allocated  to  a 
temperature  gradient  across  the  sky.  When  this  occurs,  image  comprehension  is  lowered 
because  gray  shades  that  could  have  increased  target  internal  contrast  or  other  relevant 
information  are  allocated  to  the  irrelevant  sky  temperature  gradient. 

The  purpose  of  this  Non-linear  Display  Mapping  (NDM)  research  is  to  determine  the 
level  of  image  comprehension  improvement  gained  by  allowing  the  user  to  tune  the 
sensor  display  to  the  scene.  This  is  achieved  by  the  user’s  active  control  of  gray  shade 
mapping  to  meet  his  comprehension  or  target  acquisition  needs.  Figure  1 depicts 
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al  location  of  gray  shaaes  to  expected  target  signal  region.  The  user  can  thus  employ 
subjective,  situational  and  qualitative  scene  information  to  improve  the  displayed  image. 
NDM  does  not  increase  perceived  noise  to  achieve  contrast  improvement,  as  would  be 
expected  with  analog  gain  and  level  controls.  Thus,  if  sky  temperature  gradients,  bodies 
of  water,  fires  or  hot  pavement  are  interfering  with  scene  comprehension,  the  user  may 
“tune  them  out”  by  allocating  a small  or  even  single  gray  shade  to  those  scene  features. 
As  the  scene  or  task  changes,  the  user  can  adapt  the  display  to  meet  the  current  need.  The 
user  can  also  increase  the  gray  shades  allocated  to  objects  of  interest  within  the  scene. 


Input  Signal  (intensity) 

Figure  1.  The  NDM  Concept 

4.  IMAGE  SELECTION  AND  PROCESSING 

Fifty  frame-averaged  12-bit  thermal  images  from  an  Agema  1000  longwave  imager  were 
selected  for  an  ID  experiment.  Each  image  included  one  of  nine  tracked  vehicle  targets  in 
the  foreground,  and  mid-latitude  dense  forest  in  the  background.  The  vehicles  were:  2S3, 
BMP-1,  M109A5,  M113A2,  M1IP,  M2A2,  T-62,  T-72,  and  ZSU-23/4  (see  Figure  2). 

Five  to  7 of  8 different  aspect  orientations  were  selected  for  each  vehicle  (see  Figure  3). 
This  group  of  vehicle  images  represents  a broad  range  of  discrimination  difficulty  within 
the  category  of  tracked  vehicles  for  the  identification  task.  This  vehicle  set  also  represents 
a good  sample  of  the  types  of  foreign  and  US  tracked  vehicles  in  current  use.  The  images 
were  collected  at  a range  of  400m  with  a 20  x 13  degree  field  of  view  (FOV)  and 
resolution  of  590  x 401  pixels.  The  target  in  each  image  was  centered,  obviating  the  need 
for  any  search  by  the  observer.  The  number  of  pixels  on  target  was  small,  (from  160  to 
400  pixels),  occupying  less  than  .2%  of  the  total  image  area.  It  is  important  to  note  that 
the  range  and  FOV  selected  represent  a discrimination  significantly  beyond  the  expected 
50%  probability  of  ID  (P(id)),  where  image  information  is  highly  constrained  by  blur 
(and  thus  lower  target  internal  contrast)  and  resolution  (pixels  on  target). 
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The  50  selected  images  were  processed  using  a B-kit  automatic  algorithm  emulator, 
developed  by  E-OIR  measurements  for  the  Night  Vision  Electronic  Sensors  Directorate 
(NVESD).  The  automatic  algorithm  emulator  required  no  user  input,  and  the  display 
parameters  were  determined  by  the  overall  scene  characteristics.  Concurrently,  the  same 
50  images  were  processed  with  NDM  software  developed  by  NVESD.  The  NDM 
software  user  viewed  representative  images  and  their  histograms  for  each  of  the  five 
collection  environments  from  which  images  were  drawn.  The  user  then  determined 
regions  to  be  compressed  or  enhanced.  Scene  characteristics  such  as  cold  sky  and  hot 
ground  were  suppressed  for  the  NDM  processed  images.  Images  from  the  same  location 
and  collection  date  were  processed  using  the  same  NDM  suppression  and  enhancement 
parameters  to  represent  sensor  output  optimized  for  the  scene,  rather  than  for  the  target. 
This  eliminated  any  bias  based  on  user  knowledge  of  the  specific  target  in  each  image 
and  its  critical  features  or  characteristics.  The  result  of  the  two  processing  paths  above 
yielded  a total  of  100  images  (50  B-kit  automatic  and  50  NDM)  that  were  presented  to 
the  subjects  in  a forced-choice  vehicle  ID  computer  perception  test.  This  test  software 
Was  initially  developed  by  NVESD  and  has  been  used  with  a multitude  of  image  sets  to 
conduct  over  20  different  experiments  over  the  last  four  years. 


Figure  2.  Vehicle  targets  types 
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Figure  4 is  a sample  of  the  experimental  imagery  for  each  treatment,  and  A1  and  A2  were 
generated  from  the  same  base  image.  It  is  easy  to  notice  the  relatively  large  number  of 
gray  shades  allocated  to  the  sky  in  Al,  whereas  in  A2  the  sky  is  reduced  to  a single  gray 
shade.  Note  the  greater  internal  target  contrast  of  A2,  and  the  separation  of  the  target 
silhouette  from  the  tree  line.  The  greater  target  to  background  contrast  and  internal  target 
contrast  of  A2  is  made  possible  by  the  re-allocation  of  gray  shades  from  cold  sky 
temperatures,  where  there  is  little  meaningful  information,  to  expected  target 
temperatures. 


Images  B1  and  B2  in  Figure  4 were  generated  from  a base  image  different  in  location  and 
environmental  conditions  from  Al  and  A2.  The  overcast  sky  in  B1  and  B2  is  much  closer 
to  ambient  temperature  than  in  Al  and  A2,  where  no  clouds  are  evident.  Note  that  the 
automatic  algorithm  emulator  in  B1  allocates  many  gray  shades  to  the  tree  line  but 
saturates  the  target,  revealing  little  internal  target  detail.  While  the  overall  scene  in  B2  is 
lower  contrast  than  Bl,  the  target  has  good  internal  contrast,  revealing  the  unsupported 
tracks,  the  location  of  the  engine  and  exhaust,  the  turret  crease,  the  folded  radar  dish,  and 
other  cues  that  distinguish  this  vehicle  as  a ZSU-23/4. 
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A1 . Ml  09 A5,  B-kit  emulator  A2.  Ml  09A5,  NDM 


Bl.  ZSU-23/4,  B-kit  emulator  B2.  ZSU-23/4,  NDM 


Figure  4.  Low-resolution  experimental  image  examples 

5.  EXPERIMENTAL  METHOD 

All  test  subjects  were  required  to  train  to  a 96%  medium  range  P(id)  standard  (less  than 
1500  pixels  on  target)  for  the  experimental  set  of  nine  vehicles  before  participating  in  this 
experiment.  The  US  Army’s  Recognition  of  Combat  Vehicles 2 (ROC-V)  thermal 
signature  training  software  was  used  to  achieve  this  level  of  proficiency.  Training  thermal 
signatures  is  necessary  for  thermal  ID  experiments  to  reduce  error  introduced  by 
guessing.  For  example,  if  a test  subject  is  only  30%  proficient  in  a set  of  10  vehicles  at 
close  range  (he  can  only  positively  ID  3 of  10),  70%  or  more  of  his  ID  calls  for  that  set 
may  be  guesses.  This  introduces  random  error  into  the  experimental  data  and  reduces  the 
reliability  of  results.  Further,  well-trained  subjects  can  extract  or  deduce  information 
from  an  image  that  less  well-trained  subjects  cannot. 

Subjects  were  individually  administered  the  100-image  computer  perception  test 
discussed  above  upon  completion  of  the  required  ROC-V  training.  Subjects  were 
sequentially  shown  the  images  in  randomized  order  and  required  to  select  the  most  likely 
vehicle  identity  based  on  what  they  could  discern  from  the  image.  Images  could  not  be 
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skipped  or  revisited.  No  time  limit,  per  image  or  for  the  test  overall,  was  imposed.  Users 
received  no  feedback  until  the  test’s  completion,  and  scores  or  any  other  aspects  of  the 
test  were  not  discussed  until  all  subjects  had  completed  their  tests.  Subject  test  results 
were  recorded  as  individual  files  for  each  individual.  These  files  included  subject 
identifiers,  the  image  presentation  order,  and  the  scores,  target  calls  and  time  between 
target  calls  for  each  image.  It  is  important  to  note  that  the  test  imagery  was  collected 
separately  from  the  ROC-V  training  imagery  to  avoid  artifactual  image  identification, 
rather  than  the  desired  vehicle  identification. 

6.  RESULTS 

Twenty  well-trained  thermal  sight  users  completed  NVESD  perception  test.  The  mean 
scores  for  B-kit  automatic  and  NDM  images  were  25.0  % (14%  above  chance)  and  32.0% 
(21%  above  chance)  respectively.  Respective  standard  deviations  were  .78  and  .74  (1.6% 
and  1.5%).  The  high  and  low  scores  for  the  B-kit  treatment  were  52%  and  16%,  with 
70%  of  subjects  scoring  between  16%  and  30%.  The  highest  and  lowest  NDM  treatment 
scores  were  64%  and  20%,  with  70%  of  subjects  scoring  between  24%  and  40%:  Figure  5 
illustrates  each  individual’s  ID  performance  with  each  treatment;  with  subjects  ordered 
by  B-kit  treatment  score.  The  error  bars  indicate  that  19  of  20  subjects  had  significantly 
higher  scores  for  the  NDM  treatment.  NDM  scores  for  those  who  improved  were  from 
12%  to  100%  higher. 
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7.  CONCLUSION 


If  the  treatments  were  theoretically  equal,  the  probability  of  19  of  20  observers 
scoring  higher  with  NDM  would  be  21  in  1,048,576.  There  is  thus  a better  than 
99.9%  probability  that  NDM  does  improve  comprehension  with  images  having  low 
resolution  on  target.  Further  experiments  will  measure  the  degree  to  which  NDM  cam 
be  expected  to  improve  identification  probabilities  and  reduce  cycle  criteria  for  N50.3 
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